Surveillance apparatus and recording medium recorded surveillance program

ABSTRACT

An economical surveillance apparatus robust to an environmental change includes an input unit for inputting images in a time series, a background image memory unit for storing a plurality of input images, a difference image memory unit for calculating and storing difference degree between a plurality of background images of the background image memory unit and an image inputted afresh, and a judging unit for judging an existence/absence of an invading object by utilizing the value of the difference degree of the difference image memory unit. This surveillance apparatus can automatically register a plurality of background images from a camera image. Since the background images that are no longer used are automatically erased, the drop of detection speed and detection sensitivity can be prevented.

BACKGROUND OF THE INVENTION

This invention relates to a surveillance apparatus for use in the fieldof a surveillance system and to a software program for accomplishing thesurveillance apparatus through a computer. More specifically, thepresent invention provides an economical surveillance apparatus that isrobust to changes of an imaging environment, and a software program foraccomplishing the surveillance apparatus.

Surveillance by using a camera has been conducted recently at variousplaces such as roads, railroad crossings, public facilities, servicefloors of banks, convenience stores, elevators of apartment buildings,and so forth. The object of surveillance is to prevent accidents andcrimes and to quickly take necessary counter-measures.

A surveillance system of public facilities such as electric power, gassupply, water distribution, etc, includes a plurality of surveillancecameras disposed not only indoors but also outdoors. Therefore, guardscan grasp on the real time basis the condition of each site throughmonitor televisions of a surveillance center.

However, the requirement of continuously monitoring pictures from aplurality of surveillance cameras for an extended time is likely toinvite “oversight” due to fatigue of the guards. It is thereforecustomary to employ a method that generates and reports an alarm soundonly when any invading object or any abnormal condition is detected, asa surveillance system enabling the guards to watch and confirm thecondition only when necessary. Various sensors such as infrared sensors,door sensors, temperature sensors, etc, have been used for thedetection, or a method that directly analyzes the images from thesurveillance cameras has been used.

The former has high reliability but involves a high cost because a largenumber of additional sensors must be installed besides the surveillancecameras. The latter does not require additional installation of specificsensors, hence is more economical. However, the latter is not free fromthe problem of an incorrect alarm because the camera images fluctuate atplaces where an environmental change is vigorous, such as outdoors, dueto sways of trees, rain and snow, reflection of water surfaces, and soforth.

A surveillance apparatus described in JP-A-5-14893, for example, detectsan invading object by comparing a background image generated by asmoothing processing in a time direction with an input image. When thebackground does not change. Therefore, this type of surveillanceapparatus can correctly detect the invading object. However, thesurveillance apparatus still involves in principle the problem ofexcessive detection in an environment where the background incessantlychanges, such as outdoors.

An image processing apparatus for surveillance according toJP-A-10-105839 uses a mean value of several past frames as a backgroundimage, calculates the difference between the background image and aninput image to extract an abnormal image, further calculates itsdifference from a previous frame and distinguishes an abnormal changefrom others. Because the background image is generated by the meanimage, degradation of the detection sensitivity occurs when an invadedobject is contained in the background image.

SUMMARY OF THE INVENTION

In addition to the problems described above, the conventionalsurveillance apparatuses have the problem that when reliability isimproved, the cost of production becomes higher, and when the cost islowered, an incorrect alert increases. It is therefore an object of thepresent invention to provide an economical surveillance apparatus thatis robust to an environmental change, and a software program foraccomplishing the surveillance apparatus.

To accomplish this object, the present invention provide a surveillanceapparatus including background image memory means for storing aplurality images inputted in a time series, memory means for recording adifference map, for calculating and storing a minimum value (differencedegree) of an absolute value of a difference between a plurality ofimages and an image afresh inputted, for each pixel, and judgment meansfor judging the existence/absence of an invading body by using thedifference.

These and other objects, features and advantages of the presentinvention will become more apparent in view of the following detaileddescription of the preferred embodiments in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a principle of detection of an invading object;

FIG. 2 shows a surveillance system utilizing a surveillance apparatusaccording to an embodiment of the present invention;

FIG. 3 is a system structural view of a surveillance apparatus accordingto an embodiment of the present invention;

FIG. 4 shows an example of a remote control device according to thepresent invention;

FIG. 5 shows an example of a control parameter-setting screen accordingto the present invention;

FIG. 6 shows an example of a basic screen according to the presentinvention;

FIG. 7 is a flowchart of an invading object detection control programaccording to the present invention;

FIG. 8 shows an example of a data structure of recording data accordingto the present invention;

FIG. 9 shows an example of data structures of an input image and abackground image according to the present invention;

FIG. 10 shows an example of a data structure of a difference imageaccording to the present invention;

FIG. 11 shows an example of control data according to the presentinvention;

FIGS. 12A-12D show examples of waveforms of difference degree;

FIG. 13 is a flowchart of a detection processing according to thepresent invention;

FIG. 14 is a flowchart of a block-wise change region image generationaccording to the present invention;

FIG. 15 is a flowchart of an invading object judgment processingaccording to the present invention;

FIG. 16 is a flowchart of a background image registration processingaccording to the present invention; and

FIG. 17 is a flowchart of a background image erasing processingaccording to the present invention.

DESCRIPTION OF THE EMBODIMENTS

FIG. 1 shows a principle of an invading object detection by asurveillance apparatus according to the present invention. A backgroundimage 174 in which no invading object exists is stored in a memory inadvance. This image is compared with an input image 173 to detect animage portion that has changed, and to identify whether or not anyinvading object enters the image of a surveillance camera. In thisinstance, several patterns of the background image are stored and all ofthem are compared with the input image. If there is the possibility ofthe sway of branches of background trees, for example, several patternsof the sway are stored beforehand. Even when the branches of the treessway by wind, the input image coincides with any of the backgroundimages because the patterns of the sway are stored. Though mere sway ofbranches has been likely in the past to result in the erroneous judgmentof the invading object, the present invention can accomplish robustdetection of the invading object against fluctuation or changes of thebackground.

FIG. 2 shows an example of a surveillance system utilizing asurveillance apparatus according to the present invention. An imagesignal from a surveillance camera 200 is inputted from a video inputterminal 110 of a surveillance apparatus 100 and is converted to adigital image. The digital image is displayed as an analog image on adisplay 220 from a video output terminal 110. An alarm signal generatedwhen any invading object is judged as existing is sent from a MODEMterminal 140 to a surveillance center through a telephone line. A mobilephone 210, too, can be utilized in this case. The surveillance apparatus100 is controlled by an instruction from a remote control device 400through an IR (infrared ray) receiver 130.

Conventional analog surveillance cameras and displays can be as suchutilized as surveillance equipment to be connected to the surveillanceapparatus 100 according to the present invention. Furthermore, the alarmsignals can be transferred to a remote place through the MODEM.

FIG. 3 shows an example of a system construction when the surveillanceapparatus 100 according to the present invention is achieved by using ageneral-purpose computer. In other words, the general-purpose computerexecutes a software program to accomplish the surveillance apparatus 100according to the present invention.

Reference numeral 111 in the drawing denotes an A/D converter forconverting an analog image signal 210 from a surveillance camera 200 toa digital image. The digital image so converted is inputted to a memory170 through an interface 112 and at the same time, to a video memory118.

The video memory 118 stores the image displayed on a display 220 asdigital data. Reference numeral 119 denotes a D/A converter that isgenerally called “RAMDAC”, too. This D/A converter 119 serially readsthe data written to the video memory 118 in match with a line scanningspeed and depicts an image on the display 220. Therefore, when the dataof the video memory 118 is updated, the updated content is immediatelyreflected on the display content of the display 220.

The display 220 is a device for displaying the image. Such a display maybe a compact CRT or a plasma display, for example, or a liquid crystaltype display device. Continuous moving images can be displayed when suchimages are repeatedly inputted at a frequency of about 30 times/second.

An external memory device 160 is a large-capacity memory device such asa hard disk and semi-permanently stores the digital data. The externalmemory device 160 may be of a type that can be fitted to, and removedfrom, the apparatus main body for each storage device, such as a PCMCIAsystem hard disk card, or of a type in which only a recording medium isremovable such as DVD-RAM or a flash memory.

A CPU 150 executes a software program for controlling invading objectdetection and for achieving the functions of the surveillance apparatusof the present invention. A memory 170 constantly stores the program,and stores also the data necessary for executing the program.

A modem 139 transfers recorded images and alarm information through atelephone line. An IR receiver 130 is an input device of informationfrom the remote control device. The input information is transmitted tothe CPU 150 through an interface 131 and is appropriately processed.Reference numeral 180 denotes a data bus for mutually connecting thedevices described above.

In the system construction of the surveillance apparatus describedabove, according to an invading object detection control program 172stored in the memory 170 in this embodiment the input image as inputimage date 173 is stored into the memory 170, serially transfers theimage to the video memory 118 and displays the image on the display 220.After the invading object detection processing is completed, the imageis stored as recorded image data in the external memory device 160,whenever necessary. When the invading object is judged as existing, thealarm signal is outputted through the MODEM 139.

FIG. 4 shows an example of the remote control device that givesinstructions to the surveillance apparatus according to the presentinvention. All the instruction commands from the remote control device400 are generated through IR signals from an IR emitting portion 420.Reference numeral 410 denotes a button for instructing ON/OFF of a powersource. Reference numeral 430 denotes a button for instructing detectionof the invading object. Reference numeral 432 denotes a button forsetting control parameters of the invading object detection. Referencenumeral 433 denotes a button for setting various data. Reference numeral434 denotes a paddle switch for selecting items. This button can bemoved to right and left and up and down. Reference numeral 435 denotes apaddle switch for setting numeric values of from 0 to 9. When the upperpart of this button is pushed, a large value of up to 9 as the maximumis acquired. When its lower part is pushed, a smaller value of down to 0is acquired.

Next, the mode of operation of the control parameter-setting button 432and the detection button 430 will be explained. When the controlparameter-setting button 432 is pushed, a control parameter settingscreen for the invading object detection, shown in FIG. 5, is displayedon the display 220. In FIG. 5, reference numeral 500 denotes a cursor.Reference numeral 510 denotes a background image number input region.Reference numeral 511 denotes a background expiration time input regionas a period of time necessary for erasing the registered backgrounds.Reference numeral 512 denotes an object judgment size input region.

The paddle switch 434 is operated to position the cursor to apredetermined digit of the background image number input region 510. Thepaddle switch 435 is operated to input a desired background imagenumber. Next, the cursor is moved to a setting region 530 and thesetting button 433 is pushed. In consequence, the background imagenumber that can be stored to maximum is set as a parameter.

Similarly, the paddle switch 434 is operated to position the cursor to apredetermined digit of the background expiration time input region 511.The paddle switch 435 is operated to input a desired backgroundexpiration time. Next, the cursor is moved to a setting region 531 andthe setting button 433 is pushed. In consequence, the shortestbackground expiration time is set as a parameter.

The paddle switch 434 is further operated to position the cursor to apredetermined digit of an object judgment size input region 512, and thepaddle switch 435 is operated to input a desired object judgment size.Next, the cursor is moved to a setting region 532 and the set button 433is pushed. In consequence, a minimum object judgment size is set as aparameter.

Finally, the cursor is moved to a region 540 and the set button 433 ispushed, thereby completing the control parameter setting screen.Incidentally, it is also possible to furnish the surveillance apparatusitself with the buttons and the paddle switches in place of using theremote control device. When the general-purpose computer is used toachieve the surveillance apparatus of the present invention, thefunctions of the respective buttons and paddle switches may be allocatedto the keys of the keyboard in place of the remote control device.

When the detection button 430 is pushed, the basic screen shown in FIG.6 is displayed on the display 220. In FIG. 6, reference numeral 600denotes a display region of the input image. Reference numeral 610denotes a region for tabulating and displaying the images generated bydetecting and recording the invading object. Recorded images 611 aredisplayed in a reduced scale while they are aligned. At this time, anobject existing region 612 and detection time information 613 aredisplayed, too. Reference numerals 620, 630 and 640 denote parametersthat are set at present to detect the invading object. Reference numeral620 denotes a background image number. Reference numeral 630 denotes abackground expiration time and 640 does an object judgment size.

This basic screen enables a user to quickly grasp the detectioncondition of the invading object from the past to the present.

FIG. 7 shows an example of an invading object detection control program172 for accomplishing control of the surveillance apparatus 100 of thepresent invention. A step 900 is an initialization processing. Thedisplay 220 displays the basic screen shown in FIG. 6. A step 905 resetsa variable “STATUS” for controlling a status of detection to “0”.

The following control processing is executed while a power source of thesurveillance apparatus is turned on (step 920).

A step 930 checks the condition of the buttons of the remote controldevice 400, and a step 940 executes the following judgment processing.

When the control parameter-setting button 432 is pushed, a step 941executes a control parameter-setting processing. This processing is aninteractive processing with the user. After the system displays thecontrol parameter-setting screen on the display, the user operates theremote control device 400 to input numeric values. After this processingis completed, the display content of the display returns to the basicscreen shown in FIG. 6.

Next, a judgment processing of a step 950 is executed.

When the variable “STATUS” is “0” and the detection button 430 ispushed, a detection initialization processing 951 is executed. In thisdetection initialization processing, recording start year/month/day, arecording start time and a number of recording N (a number of recordedframes), that are the header information of recording data when theimage containing the invading object is recorded, are written into theexternal memory device 160. The recording number is “0” at first. Aframe number of the input image is initialized to “1”, and the image isinputted from the surveillance camera and is registered as backgroundimage data 174. Control data 176 for detection is set to the initialstate, too. A value “1” is set to the variable “STATUS” in step 952 toset the mode to a detection mode.

When the variable “STATUS” is “1” and the detection button 430 ispushed, a step 953 is executed. The variable “STATUS” is reset to “0”and the mode is brought to the detection finish state.

In a judgment processing of step 960, the detection process is conductedin a step 961 when STATUS is “1”, and when any invading object isdetected, the image is written into the external memory device 160 andthe alarm signal is outputted. The detail of this detection processingwill be explained later with reference to FIG. 13. Therefore, variousdata structures used for the detection processing will be firstexplained.

FIG. 8 shows an example of the data structure of the recorded data 161in the present invention. The recorded data includes a header portionand an image portion. A recording start year/month/day 161-1, arecording start time 161-2 and the number of recording (a number ofrecorded frames) 161-3 are recorded to the header portion. A framenumber 161-4, an image acquisition year/month/day 161-5, an imageacquisition time 161-6 and image data 161-7 are recorded to the imageportion.

FIG. 9 shows the data structures of the input image and the backgroundimage. Reference numeral 173 denotes the input image. The input imagehas W pixels in the transverse direction and H pixels in thelongitudinal direction. In the images of the NTSC standard, W is 640 andH is 480. Reference numeral 174 denotes the background image. In thebackground image 174-2, maximum K pixel values are registered for eachpixel at an (i, j) position. Background registration number-of-timesarray 174-1 is an array for storing the number of times of registrationof the background pixel value for each pixel.

The data structure further includes a frame number array 174-3 at thetime of registration, for storing reference history information as thebackground image for each pixel and the latest referred frame numberarray 174-4 that is looked up. The frame number array at the time ofregistration is an array for storing the frame number of the pixelvalues registered. In other words, it manages the time at which thebackground images are registered.

The latest referred frame number array is an array for storing the framenumber of the pixel value that is looked up at the latest. In otherwords, it manages the time at which it is used lastly as the backgroundimage.

In the present invention, reference history information of thebackground image is stored for each pixel. Therefore, registration anderasing of the background image can be finely conducted in a pixel unit.In other words, under an imaging condition where only a part on thescreen periodically changes, only the changing part can be additionallyregistered or erased as the background portion.

FIG. 10 shows a data structure of a difference image comprising aminimum value (difference degree) of an absolute value of differencebetween the input image and a plurality of background images for eachpixel. Reference numeral 175-1 denotes an image comprising a minimumvalue determined by calculating a value, at which the absolute value ofthe difference between a plurality of background images and the inputimage becomes minimal, for each pixel. The difference between thebackground image and the input image can be known in a pixel unit fromthis image. Reference numeral 175-1-1 denotes a minimum backgroundnumber array for storing a background pixel number when the absolutevalue of difference is minimal. It becomes possible to know from thecontent of the minimum value background number array which backgroundpixel is most analogous to the input pixel. When this minimum valuebackground number array is used, reference history information of thebackground pixels can be simply updated.

Reference numeral 175-2 denotes a block-wise mean value image thatstores a mean value of values inside each block when the minimum valueimage 175-1 is divided into Q transverse blocks and R longitudinalblocks. In this embodiment, Q is 16 and R is 12, but these values arenot limitative. When the mean value is determined block-wise, thedifference between the background image and the input image can be knownin a block unit. Since the invading object has a certain size, detectionperformance becomes more robust by judging a rough difference in a blockunit.

Reference numeral 175-3 denotes a block-wise change region imageobtained by a processing which sets the block-wise mean value image175-2 to “1” when it is greater than a predetermined threshold value andto “0” at other times. The block detected in this way has the value “1”.

FIG. 11 denotes a structure of control data for detection. Referencenumeral 176-1 denotes a FIFO (Fast-In Fast-Out) type shift buffer forstoring the block-wise mean value images for past M frames. Thisstructure is looked up when the change of the difference for each blockis analyzed in a time series. Reference numeral 176-2 denotes a FIFO(Fast-In Fast-Out) type shift buffer for storing the block-wise changeregion images for each block for past M frames.

Reference numeral 176-3 denotes a block-wise interconnection image forallocating the same label for distinguishing individual interconnectioncomponents, for each connection component having the value “1” of theblock-wise change region image 175-3. Reference numeral 176-4 denotes afeature quantity table that stores upper left coordinates (XS, YS) andlower right coordinates (XE, YE) of a circumscribing rectangle and itsarea for each label of the block-wise interconnection image 176-3. Finaljudgment as to whether the object is the invading object or otherobjects depend on whether or not this feature quantity coincides with apredetermined standard.

FIGS. 12A-12D show examples of time waveforms of the difference degreeof each block in the present invention. FIG. 12A shows an example whereno structural change of the background exists. In this case, thedifference degree having small values continue. FIG. 12B shows anexample where a structural change of the background exists. In thisexample, the difference degree have high values, and a flat conditioncontinues. The case where the structural change of the background existsincludes the case where a car comes to and parks at a place undersurveillance, for example. The car is not a part of the background atthe beginning but becomes a part after it parks.

When the background entirely changes from the previous state, thepresent invention registers afresh the input image as the backgroundimage. FIG. 12C shows an example where the invading object exists, anddifference degree reaches a high value only for a short time. This isthe case where a person passes by the place under surveillance, forexample. Detecting blocks exhibiting such a change, the presentinvention judges the number of times of the changes in this block andspatial expansion of the changing block, and classifies the change intothe periodic change of the background and the true invading object.

FIG. 12D shows an example of a waveform of a block change when thebackground changes periodically due to influences of wind, and so forth.In such a case, a large number of changes occur within a short time.Detecting such a block, the present invention judges this block as thechanging background and registers the input image of this block portionas a new background for each pixel.

Even when the surveillance camera images swaying trees or reflection ofthe water surface, the function of the present invention for registeringa plurality of backgrounds for each pixel can eliminate incorrectjudgment of such trees or reflection as the invading object andcontinuous generation of the alarm. Incidentally, the range δ of thepast reference time is 2 seconds, that is, 60 frames (M=60), but thisvalue is not restrictive.

FIG. 13 is a flowchart showing the detail of the detection processing961 in the present invention.

First, in a step 1000, the image from the surveillance camera 200 isinputted and the image as the input image 173 is stored in the memory170.

In a next step 1010, an absolute value of the difference between theinput image 173 and the background image 174-2 is calculated for eachpixel, then calculates a minimum value among all the background imagesis calculated and a minimum value image 175-1 is generated. In thisstep, the number of the background image assuming the minimum value isstored to a minimum value background number array 175-1-1.

In a step 1020, a mean value of the value of the minimum value image175-1 is calculated for each block and a block-wise mean value image175-2 is generated. The block-wise mean value image 175-2 thus generatedis transferred further to a shift buffer 176-1 in a step 1030.

In a next step 1040, a block-wise change region image 175-2 isgenerated. The detail of this step will be explained later withreference to FIG. 14. In a step 1050, the block-wise change region image175-2 is transferred to a shift buffer 176-2. On the basis of the datagenerated by the process steps described above, an invading objectjudgment processing is executed in a step 1060, a background imageregistration processing is executed in a step 1070 and a backgroundimage erase processing is executed in a step 1080. Finally, in a step1090, the frame number is incremented by one.

Even when any change occurs in the background during imaging by thesurveillance camera, the process steps of the present inventiondescribed above automatically update the background and can stablydetect only the invading object.

The steps 1040, 1060, 1070 and 1080 will be explained in further detailwith reference to FIGS. 14, 15, 16 and 17.

FIG. 14 is a flowchart showing the detail of a block-wise change regiongeneration processing. This embodiment divides the whole screen intoQ×R=16×12 blocks. When a large body is to be detected, however, thisdivision number may be smaller. The block-wise change region generationprocessing can be achieved when the following steps are executed for allthe blocks.

First, a variable “LIPPLE” and a variable “HEIGHT”, each for judgingwhether or not each block changes, are calculated. In a step 1110, thedifference between a maximum value and a minimum value of a block (i, j)inside a shift zone of the shift buffer 176-1 storing the block-wisemean value image is calculated, and the difference is stored in thevariable “LIPPLE”. Next, the value of the block (i, j) of the block-wisemean value image 175-2 of the present frame is stored in the variable“HEIGHT”.

In a step 1120, it is judged that whether or not the variable “HEIGHT”(block mean value of present frame) is greater than a predeterminedthreshold value th1 and also the variable “LIPPLE” (the differencebetween the maximum value and the minimum value of the block mean valueof the present frame) is greater than a predetermined threshold valueth2. When both variables are greater than the respective thresholdvalues, the state is judged as the state shown in FIG. 12C. In a step1125, “1” is set to the block (i, j) of the block-wise change regionimage 175-3. When both are smaller, on the other hand, in a step 1130“0” is set to the block (i, j) of the block-wise change region image175-3. In this embodiment, both th1 and th2 are 5 but other values maybe used. Incidentally, the range of the value the pixels take is from 0to 255.

In a step 1132, it is judged whether or not the variable “HEIGHT” issmaller than the predetermined threshold value th1 and the variable“LIPPLE” is smaller than the predetermined threshold value th2. If bothvariables are smaller than the respective threshold values, the state isjudged as the state shown in FIG. 12A, and an update processing of thereference history information of the background image from the steps1135 to 1155 is executed. In the step 1135, the range of the address ofthe block (i, j) on the practical image is calculated. It also updatesthe reference history information of the background pixel that is lookedup.

First, in a step 1150, the value of the point (i2, j2) of the minimumvalue background number array 175-1-1 is set to the variable k. Next,the point (i2, j2, k) of the array 174-4 recording the latest framenumber that is looked up in the step 1150 is updated to the presentframe number. In this way, this embodiment updates the reference historyinformation of the background pixels inside all the corresponding blockswhen the image inputted at present coincides with the background imageas a block.

FIG. 15 is a flowchart showing the detail of a judgment processing forjudging the existence/absence of the invading object.

First, the feature quantities necessary for detecting the region of theinvading object, that is, the coordinates of a circumscribed rectangleencompassing the invading object and the area of the rectangle, aredetermined. Therefore, as to the blocks having a value “1” of theblock-wise change region image 175-3, in a step 1200, it is judged thatthese blocks as the same interconnection component if their left, right,upper or lower block is “1”, the same label is provided to them and theblock-wise interconnection image 176-3 is generated. In a step 1205, acircumscribed rectangle and its area for each interconnection componenthaving the same label of the block-wise interconnected image 176-3 aredetermined and stored in the feature quantity table 176-4.

Next, in steps 1210 to 1230, the existence/absence of the invadingobject is examined. First, in the step 1210, a variable “FLAG” is resetto “0”. When this variable “FLAG” changes to “1” in the subsequentsteps, the invading object is judged as existing. The steps 1220 to 1230are repeated for all the labels. In the step 1220, the coordinates ofthe circumscribed rectangle of the label i of the feature quantity tableand its area are read out and set to variables XS, XE, YS, YE and AREA.In the step 1225, it is judged whether conditions(XE−XS+1)×(YE−YS+1)>th4 and AREA>th5 are satisfied. When theseconditions are satisfied, in the step 1230, the variable FLAG is set to“1”.

This embodiment judges both the size of the circumscribed rectangle andits area for the following reason. Since the properties of the shape ofthe invading object such as people are different from those of theinvading object such as the swaying branches of trees, both of thefeatures are employed for distinguishing their differences. Here, theconcrete values of the threshold values th4 and th5 vary depending onthe invading object to be detected. Therefore, the user sets in advancesuitable values. In this embodiment, both th4 and th5 are 2.

Finally, in the step 1235, it is judged whether or not the variable“FLAG” is “1”. When it is “1”, the process at the time of existence ofthe invading object in the steps 1240 to 1255 is executed. In the step1240, the recording number (the number of recorded frames) 161-3 isincremented by one. In the step 1245, the present input image 173 withthe frame number, the image acquisition year/month/day and the imageacquisition time as the recording data are written into the externalmemory device 160. Next, in the step 1250, the size of the input image173 and displays the image on the display 220 is reduced. In thisinstance, the circumscribed rectangle encompassing the invading objectmay be overwritten and displayed in the image. The detection time mayfurther be displayed. When display is made in this way, the user caninstantaneously grasp the detection state of the invading object fromthe past to the present moment. In the step 1255, the alarm signal isoutput through the modem 140.

FIG. 16 is a flowchart showing in detail the background imageregistration processing. The background image registration processingcan be achieved by executing the following process for all the blocks.First, the variables “LIPPLE”, “HEIGHT”, and “SUM” are calculated tojudge whether or not each block changes. In a step 1310, the differencebetween the maximum value and the minimum value of the block (i, j)inside the shift zone of the shift buffer 176-1 storing the block-wisemean value image is calculated and stored in the variable “LIPPLE”.Next, in a step 1315, the mean value of the block (i, j) inside theshift zone of the shift buffer 176-1 is calculated and stored in thevariable “HEIGHT”.

In a step 1320, the sum of the values of the blocks (i, j) inside theshift zone of the shift buffer 176-2 storing the block-wise changeregion image is calculated and stored in the variable “SUM”.

In a next step 1325, a registration condition of the background image isjudged. The registration condition includes two conditions. The firstone is “HEIGHT”>th1 and “LIPPLE”<th2, and this is the case where thecondition of FIG. 12B is detected. The second one is “SUM”>th3, and thisis the case where the condition of FIG. 12D is detected. The formerrepresents the structural change of the background, and corresponds tothe case where a car parks afresh inside the background, for example.The latter represents the periodic change of the background andcorresponds to the case where branches of trees incessantly sway due toa strong wind.

When such conditions are detected, steps 1230 to 1370 are executed toafresh register the background image. Here, th3 is 30 in thisembodiment. This is the value when the past reference time range δ is 2seconds, that is, M=60, and represents detection of the condition wherethe background has changed more than 50% in the past.

In a step 1330, the range of the addresses on the practical images ofthe block (i, j) is calculated. Only when the difference degree of thepixels inside the block is excessively large, the pixel value of theinput image is partially registered as a background image for eachpixel.

First, in a step 1345, the value of a point (i2, j2) of the minimumvalue image 175-1 of the absolute value of the difference is set to avariable “DIFF”. Next, in a step 1350, it is judges whether or not thevariable “DIFF” is greater than th1. When it is greater, the pixels ofthe input image are different from all the background pixels that areregistered at present. Therefore, steps 1355 to 1370 are executed toregister the background pixels.

In the step 1355, the number of registration portion of an emptybackground of the point (i2, j2) of the background image 174-2 is set toa variable k. Next, in the step 1360, the pixel value of the point (i2,j2) of the input image 173 is registered afresh to the point (i2, j2, k)of the background image 174-2. In the step 1365, the value of the point(i2, j2) of the array 174-1 recording the number of times of backgroundregistration is incremented by one. Finally, in the step 1370, thepresent frame number is written in the frame number array 174-3 at thetime of registration and to the point (i2, j2, k) of the latest framenumber array 174-4 that is looked up.

FIG. 17 is a flowchart that shows in detail a background image eraseprocessing. Contrary to the processing shown in FIG. 16 for dynamicallyregistering the background image, this processing dynamically erases thebackground images that are not used. This processing reduces the memorycapacity necessary for the apparatus and shortens the computation timenecessary for detecting the invading object. Further, this processingcan prevent deterioration of detection sensitivity that occurs when agreat number of unnecessary background images are stored.

First, steps 1410 to 1450 are executed for all the background images. Inthe step 1410, the value of the point (i,j) of the backgroundregistration number-of-times array 174-1 is set to the variable n. Thefollowing steps are executed for the variable k from 1 to n times.

First, in the step 1420, it is judged whether or not the value of thepoint (i, j, k) of the frame number array at the time of registration isgreater than 1. When it is greater than 1, the background image isjudged as being not yet erased, and the processing of the steps 1425 to1450 is executed. In the step 1425, the difference between the value ofthe point (i, j, k) of the latest frame number array 174-4 that islooked up and the value of the point (i, j, k) of the frame number array174-3 at the time of registration is calculated and stored in thevariable “EVAL”. The value of this variable “EVAL” represents the periodin which the corresponding background pixel is looked up. In the step1430, it is judged whether or not the variable “EVAL” is smaller than apredetermined threshold value th6. When the variable “EVAL” is smaller,the threshold value th6 is written to the variable “EVAL”. Here, thethreshold value th6 can take an arbitrary value. When a large value isset, the background can be stored for a long time. In this embodiment,th6 is set to a value of the number of frames corresponding to about 10seconds, that is, 300. The reason why the threshold value th6 is writtento the variable “EVAL” when the variable “EVAL” is smaller than thepredetermined threshold value th6 is to prevent the background having ashort reference time from being immediately erased.

Next, in the step 1440, the difference between the present frame numberand the value of the point (i, j, k) of the latest frame number array174-4 that is looked up is calculated and stored in a variable “PERIOD”.This variable “PERIOD” represents the period that has not been looked upso far.

Importance of the present point is judged from a function using the twovariables, i.e. the period “EVAL” that is looked up and the period“PERIOD” that is not looked up, and whether or not the background pixelsare to be erased is decided. In this embodiment, the step 1445 executesthis judgment by judging whether or not (PERIOD-EVAL) is positive, butwhether or not (PERIOD-EVAL/K) is positive may be used as anotherjudgment reference. Here, K is a constant. When judgment represents thatbackground pixels should be erased, in the step 1450, the value of thepoint (i, j, k) of the frame number array 174-3 at the time ofregistration is reset to “0”. In consequence, the correspondingbackground pixel is erased.

Since the present invention can detect the invading object from theimage of the surveillance camera, the present invention does not requireany additional sensors besides the surveillance camera and can thereforeprovide an economical surveillance apparatus. Because a plurality ofbackground images can be automatically registered from the camera imagesfrom the place such as outdoors where the environmental change isvigorous, the present invention can provide a robust surveillanceapparatus that does not generate an incorrect report even when theenvironmental change occurs due to the sway of trees, rain and water,reflection from the water surface, and so forth. Furthermore, becausethe background images that are not used are automatically erased, thepresent invention can prevent the drop of the detection speed and thedetection sensitivity.

While the present invention has been described above in conjunction withthe preferred embodiments, one of ordinary skill in the art could beenabled by this disclosure to make various modifications to theembodiments and still be within the scope and spirit of the invention asdefined in the appended claims.

What is claimed is:
 1. A surveillance apparatus comprising: input meansfor inputting images in a time series; background image memory means forregistering a plurality of different background images from said images;difference image memory means for calculating and storing a minimumvalue of a difference between each of said plurality of differentbackground images and an image inputted afresh for each pixel; andjudgment means for judging an existence/absence of an invading object onthe basis of said difference.
 2. A surveillance apparatus according toclaim 1, wherein each of said plurality of different background imagescomprises a single or a plurality of pixel values registered for eachpixel.
 3. A surveillance apparatus according to claim 2, wherein: eachbackground image is registered or erased for each pixel; erasing isdetermined on the basis of a period from a registration time of saidpixel to a time at which said pixel is last looked up, and a period froma time at which said pixel is last looked up to a present time; whenregistration is made, a new pixel value is written to an empty region ifsaid empty region exists in said background image memory means, and ifsaid empty region does not exist, importance of each pixel value of acorresponding pixel is calculated and is overwritten to an image valuehaving the lowest degree of importance.
 4. A surveillance apparatusaccording to claim 1, further comprising: memory means for recordingsaid image inputted afresh for each pixel, when said judgment meansjudges that an invading object exists.
 5. A surveillance apparatusaccording to claim 1, wherein: said difference image memory meansincludes minimum value image memory means for storing a minimum valueimage comprising an absolute value of a difference for each pixel, andblock-wise mean image memory means for dividing said minimum value imageinto blocks, and calculating and storing a mean value of an absolutevalue of a difference for each block; and said judgment means judges anexistence/absence of an invading object on the basis of a time serieschange of said mean value.
 6. A surveillance system comprising: one ormore cameras arranged at different designated locations to captureimages in a time series; a display unit arranged to provide a visualdisplay of the images from one or more cameras; and a surveillanceapparatus arranged to detect and analyze the images from one or morecameras, said surveillance apparatus comprising: a memory device tostore a surveillance program, and to register in advance a plurality ofdifferent background images; and a controller configured to execute thesurveillance program to detect a presence/absence of an invading objectfrom the images of one or more cameras, by determining an absolute valueof a difference between an input image and each of said plurality ofdifferent background images for each pixel, determining a minimum valueimage among all the background images at which the difference betweenthe input image and each of said plurality of different backgroundimages is minimal, determining a mean value of a difference for eachdiscrete block within the minimum value image, and determining thepresence/absence of an invading object on the basis of a time serieschange of said mean value.
 7. A surveillance system according to claim6, wherein each of said plurality of different background imagescomprises a single or a plurality of pixel values registered for eachpixel.
 8. A surveillance system according to claim 6, wherein: eachbackground image is registered or erased for each pixel; erasing isdetermined on the basis of a period from a registration time of saidpixel to a time at which said pixel is last looked up, and a period froma time at which said pixel is last looked up to a current time; and whenregistration is made, a new pixel value is written to an empty region ifsaid empty region exists in a background image memory section of saidmemory device, and if said empty region does not exist, importance ofeach pixel value of a corresponding pixel is calculated and isoverwritten to an image value having the lowest degree of importance. 9.A surveillance system according to claim 6, wherein said memory devicecomprises an input image section for recording an input image afresh foreach pixel, when said controller determines that an invading object ispresent.
 10. A surveillance system according to claim 6, wherein saidmemory device comprises: an input image storage section for storing aninput image afresh for each pixel; a background image storage sectionfor registering a plurality of different background images; and adifference image storage section for storing the minimum value image indiscrete blocks, and for storing the mean value of the difference foreach block.
 11. A surveillance apparatus comprising: an input device toinput images in a time series; a memory device to store a surveillanceprogram, and to register in advance a plurality of different backgroundimages; and a controller configured to execute the surveillance programto detect a presence/absence of an invading object from the images ofone or more cameras, by: determining an absolute value of a differencebetween an input image and each of said plurality of differentbackground images for each pixel, determining a minimum value imageamong all the background images at which the difference between theinput image and each of said plurality of different background images isminimal, determining a mean value of a difference for each discreteblock within the minimum value image, and determining thepresence/absence of an invading object on the basis of a time serieschange of said mean value.
 12. A surveillance apparatus according toclaim 11, wherein each of said plurality of different background imagescomprises a single or a plurality of pixel values registered for eachpixel.
 13. A surveillance apparatus according to claim 11, wherein: eachbackground image is registered or erased for each pixel; erasing isdetermined on the basis of a period from a registration time of saidpixel to a time at which said pixel is last looked up, and a period froma time at which said pixel is last looked up to a current time; and whenregistration is made, a new pixel value is written to an empty region ifsaid empty region exists in a background image memory section of saidmemory device, and if said empty region does not exist, importance ofeach pixel value of a corresponding pixel is calculated and isoverwritten to an image value having the lowest degree of importance.14. A surveillance apparatus according to claim 11, wherein said memorydevice comprises an input image section for recording an input imageafresh for each pixel, when said controller determines that an invadingobject is present.
 15. A surveillance apparatus according to claim 11,wherein said memory device comprises: an input image storage section forstoring an input image afresh for each pixel; a background image storagesection for registering a plurality of different background images; anda difference image storage section for storing the minimum value imagein discrete blocks, and for storing the mean value of the difference foreach block.
 16. A surveillance apparatus comprising: an input device toinput images; a memory device to store in advance a plurality ofdifferent background images; and a controller configured to compare animage input afresh from the input device with each of said plurality ofdifferent background images stored in the memory device to obtain aminimum value of a difference between an input image and each of saidplurality of different background images, and to determine whether aninvading object is present in the input image base on the minimum valueof the difference between the input image and each of said plurality ofdifferent background images, wherein said memory device comprises aninput image storage section to store the image input afresh from theinput device, a background image storage section to register theplurality of different background images, and a difference image storagesection to store the minimum value of the difference between the inputimage and each of said plurality of different background images.
 17. Asurveillance apparatus according to claim 16, wherein each of saidplurality of different background images comprises a single or aplurality of pixel values registered for each pixel.